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The new Federal Aviation Administration (FAA) Small Unmanned Aircraft rule (Part 
107) marks the first national regulations for commercial operation of small unmanned 
aircraft systems (sUAS) under 55 pounds within the National Airspace System (NAS). 
Although sUAS flights may not be performed beyond visual line-of-sight or over non- 
participant structures and people, safety of sUAS operations must still be maintained 
and tracked at all times. Moreover, future safety-critical operation of sUAS (e.g., for 
package delivery) are already being conceived and tested. NASA’s Unmanned Aircraft 
System Traffic Management (UTM) concept aims to facilitate the safe use of low-altitude 
airspace for sUAS operations. This paper introduces the UTM Risk Assessment Framework 
(URAF) which was developed to provide real-time safety evaluation and tracking capability 
within the UTM concept. The URAF uses Bayesian Belief Networks (BBNs) to propagate 
off-nominal condition probabilities based on real-time component failure indicators. This 
information is then used to assess the risk to people on the ground by calculating the 
potential impact area and the effects of the impact. The visual representation of the 
expected area of impact and the nominal risk level can assist operators and controllers 
with dynamic trajectory planning and execution. The URAF was applied to a case study 
to illustrate the concept. 


Nomenclature 
Ae Casualty Area 
Cp Drag Coefficient 
E. Expected Casualty 
ay Impact Angle 
Hy Height of Person 
Pr Probability of Impact 
Daag UAV Length 
Ry Radius of Person 
p Air Density 
Ppop Population Density 
S Reference Vehicle Area 
WwW Vehicle Weight 
Wspan Wing span 
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I. Introduction 


|W eeeecige aircraft systems (UAS) have been the focus of many commercial and civilian applications 
including infrastructure monitoring, delivery of goods, precision agriculture, public safety, search and 
rescue, disaster relief, weather monitoring, among many others. However, the implementation of such com- 
mercial operations has been slow. The large-scale small UAS (sUAS) applications are hampered by lack of 
airspace operations requirements, procedures, and support functions! as well as privacy considerations, pub- 
lic acceptance, and environmental concerns. As the demand for sUAS is projected to increase dramatically, 
the Federal Aviation Administration (FAA) published Small Unmanned Aircraft Regulations (Part 107)? 
that describe the operating requirements for commercial drone use. The Part 107 regulations state that 
unmanned aircraft weighing less than 55 lb can operate within visual line of sight (VLOS) and below 400 
feet above ground; however, operations are prohibited over populated areas. Contrary to these regulations, 
many commercial use cases necessitate beyond visual line of sight (BVLOS) operations over the general 
population to achieve full benefits (e.g., infrastructure inspection or parcel delivery). In order to enable the 
path to VLOS and BVLOS autonomous vehicle operations, NASA initiated the UAS Traffic Management 
(UTM) project. The UTM concept is a systematic approach to safely accommodate all manned and future 
UAS operations within low-altitude airspace by providing services like airspace design, corridors, dynamic 
geo-fencing, severe weather and wind avoidance, congestion management, terrain avoidance, route planning, 
re-routing, separation management, sequencing, spacing, and contingency management.? NASA collaborates 
with industry, the FAA, and several other government agencies to define roles and responsibilities of primary 
entities within the UTM ecosystem. The acquired capabilities are tested and showcased via Technology 
Capability Level (TCL) flight demonstrations. 

Within the UTM project, the UTM Safety & Risk Group based at NASA Langley Research Center 
(LaRC) is focused on the determination of potential current and future safety hazards, the development of 
UTM safety case and test scenarios, the formulation of off-nominal trajectory and impact point prediction 
methods, as well as the development of a real-time risk assessment approach in order to satisfy UTM’s 
safe airspace operations and public safety requirements. This paper addresses the need for a real-time risk 
assessment approach in UTM by providing a real-time risk assessment framework that quantifies the risks 
to bystanders due to sUAS especially for operations in populated areas. 

This research was built upon the well-studied subject of third-party risk associated with UAS operations. 
Third party risk research considers the risk to the people on the ground, who are not directly involved 
with the aircraft operation. Within the same context, Lazatint and Lum & Waggoner? estimated insurance 
liability for various UAS use cases based on fatalities due to ground and mid-air collisions by employing 
population density, population distribution, and sheltering factors. Similarly, Aalmoes et al.® developed a 
conceptual third party risk model that estimates the potential impact area and the consequence area. Using 
a different approach, Clothier et al.’ assigned various UAS type categories based on kinetic energy and 
impact severity (injury or fatalities). Lum et al. and Ford & McEntee® simulated the impact area based on 
initial failure location and used satellite imagery and census information to estimate the number of collisions 
with a bystander per flight hour. Additionally, Dalamagkidis et al.,1° Weibel & Hansman,'! and Burke 
et al.!? employed similar methods to estimate casualties and approximate the effects of kinetic energy to 
drive the target ground impact frequency based on population density. Finally, Melynk et al.!? compared 
past studies, utilized similar methods to enable casualty prediction, and also attempted to validate their 
model using fatality rates caused by general aviation crashes, obtained from historical data. The literature 
review also revealed that a dynamic methodology, capable of providing real-time risk to the population on 
the ground, had not yet been attempted. 

In order to assist UTM safety goals, this research proposes a modular methodology, named UTM Risk 
Assessment Framework (URAF), that can provide risk metrics associated with casualties in real-time. The 
novel approach adopted in this research includes the use of dynamic aircraft health and environmental data 
to provide real-time mishap likelihood by using Bayesian Belief Networks (BBNs). Section II provides an 
overview of the URAF components while Section HI presents a simulated case study to demonstrate the 
framework capabilities. Future work and conclusions are given in Sections IV and V, respectively. 
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II. UTM Risk Assessment Framework (URAF) 


Given their flexibility and envisioned use cases, small unmanned aircraft are expected to operate within 
close proximity to people and property on the ground. The proposed risk assessment framework provides 
the foundation for a tool that can be used to estimate risk associated with sUAS flights. The resulting 
tool is envisioned to assist the operators or traffic controllers by displaying real-time risk based on actual 
aircraft status and operating environment. The framework is comprised of various sections that estimate and 
display the operational risk, which is comprised of the likelihood of undesirable events and their impacts, as 


highlighted in Fig. 1. 
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Figure 1. URAF Architecture Components. 


In order to obtain real-time flight risk, the framework requires dynamic data from internal and external 
parameters, where available. These parameters include aircraft system status data provided by on-board 
health monitoring, environmental variables such as wind or other factors obtained by aircraft sensors and/or 
ground services, and population density and distribution, which are essential to estimate casualty values 
given the aircraft’s ground collision. Several commercial off-the-shelf (COTS) flight controller hardware in 
the market are capable of accessing and transmitting vital aircraft health status along with wind information 
to the ground control station. Additionally, it may be feasible to obtain dynamic population density and 
distribution by considering cellular network tower data as proxy. Once the data stated above are gathered 
on the ground control station, they are utilized by three separate modules; where 


e The Probabilistic Graphical Model estimates the likelihood of predetermined mishaps using data from 
aircraft health systems status as part of the aircraft telemetry downlink, 


e The Off-Nominal Trajectory and Impact Point Prediction module provides an expected impact point 
for the modeled mishap with an associated uncertainty bound, and 


e The Casualty Estimation module employs the impact point and associated uncertainty as well as 
population distribution data to evaluate injury or fatality to humans on the ground. 


These modules work in concert to continuously evaluate the likelihood and the consequences of the designated 
mishaps, effectively providing the real-time risk severity values, which are periodically updated and displayed 
on the ground control station computer. It is important to note that, thanks to its modular structure, the 
URAF is envisioned to be the backbone for an upcoming suite of tools, built upon each other with increasing 
fidelity and capabilities. The next sections provide further details on these modules. 


A. Probabilistic Graphical Model 


In order to examine the safety implications of operating sUAS within the UTM operational environment, 
adequate representations of mishaps and their causal and contributing factors need to be developed. As 
previously noted, the Probabilistic Graphical Model provides the likelihood of mishap occurrence by fusing 
real-time aircraft system data with detailed mishap models. To develop these models, an analysis of current 
and future sUAS operations was performed by reviewing UAS mishap reports and future use cases, as 
highlighted by Belcastro et al.? Identified hazards as well as their causal and contributing factors were then 
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used to construct the event sequence diagrams (ESDs), which are converted to probabilistic models. The 
Bayesian approach was chosen to graphically represent and model the mishaps considered within the URAF 
framework. 


1. Bayesian Approach 


The URAF employs the flexible, probabilistic approach of Bayesian belief networks (BBNs) to model and 
estimate the probability of the predetermined UAS mishaps. The BBN is a directed acyclic graphical 
representation of a framework where causal factors and outcomes are linked to each other to form a network. 
Each element of the graphical structure is associated with an initial probability distribution, acquired either 
empirically or subjectively.14 '° 

One of the major advantages of using a Bayesian approach to interpret probability is that it allows 
modelers to revise and adjust predictions of the causal factors and observe the outcomes in the light of new 
data.'® In other words, the initial subjective assumptions can be replaced by perceived evidence and the 
network will output the updated mishap probability given the presence of such evidence. Within the URAF, 
this capability allows the insertion of the real-time aircraft health monitoring data to the model and the 
revision of the likelihood of mishap occurrence. 

Another advantage of using the Bayesian approach is that the models are capable of taking into consider- 
ation both objective and subjective measures of uncertainty.'”!8 Compared to their manned counterparts, 
input data for UAS operation safety assessments contain significant sources of uncertainty. These uncertain- 
ties include: a) the lack of UAS operational data and experience, b) use of non-regulated COTS components, 
and c) a vast range of UAS configurations (multi-rotor, fixed wing, and hybrid setups), which present chal- 
lenges to identification, collection, and assessment of aircraft failure modes and their impacts. Consequently, 
Bayesian methods’ ability to operate with subjective data due to insufficient operational experience render 
the approach well-suited for quantitative UAS modeling. The Bayesian approach has also been extensively 
used to represent and evaluate uncertainty in aviation safety applications in addition to a large number of 
other complex reliable systems, such as the nuclear power industry, cyber security, space launch industry, 
etc.1419 Assessing the operational risk of flying unmanned aircraft over populated areas requires a sys- 
tems perspective approach that considers the system to be a series of interconnected and interacting parts, 
together delivering some desirable outcome and unintended consequences. !® 


2. Model Development 


As discussed above, within URAF, the probabilistic representation of undesirable events (e.g., CFIT, MAC, 
uncontrolled crash, etc.) is performed by BBNs. The model development process was initiated with the 
identification of undesirable events, respective causal factors, and their dependencies. Next, aircraft system 
failures, their propagation patterns, and combined effects of multiple failures were constructed using event 
trees, fault trees and/or failure modes & effects analysis (FMEA), all of which can be replicated using 
the BBNs.!® For this research, the Hugin Developer v.8.42° was selected as the BBN modeling software 
since it allows application program interface (API) for C, C++, Java, and .Net. The program interface 
capability enables data flow between the aircraft telemetry stream and the other complementing modules 
(i.e., Impact Point Predictor given in Section II.B, and Casualty Estimation Module provided in Section 
II.C). Additionally, Hugin software capabilities and suitability for aviation safety research were previously 
highlighted in Ancel et al.,'* Luxhoj,!° and Ancel and Shih.?4 

The hierarchy within the Bayesian networks is expressed in a manner similar to a family tree where child 
node probabilities are influenced directly by those of its parent nodes. The probability of each child node 
and the effects of the parent nodes are represented via a conditional probability table (CPT).‘*!° Barr et 
al.,?? which focused on developing a preliminary risk assessment approach, highlights a generic UAS mishap 
BBN development and data population processes in detail. 


3. Data Population 


Following the development of the undesirable event model, the next step involves the collection and pop- 
ulation of the data. As highlighted by Aalmoes et al.,° given that unmanned systems are in their early 
development stages, operational data needed to populate third party casualty models are lacking. Conse- 
quently, the data necessary to drive numerous risk models were approximated through various sources such 
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6, 23 4,6,11 


as numerical distributions, manned aircraft failure rates, and/or military unmanned or manned op- 
erational data.‘° The authors believe that, although such assumptions might be suitable for approximations 
or comparative studies of various future operational concepts, reliability data from manned aircraft oper- 
ations do not provide an adequate baseline due to vast redundancy and configuration differences between 
manned and unmanned aircraft as well as the varying operational context (military vs. civilian aircraft). 
Additionally, limited operational and empirical sUAS failure data mandates the utilization of expert pro- 
vided subjective values for conditional probability assessment. By utilizing dynamic aircraft health data, the 
methodology suggested in this paper provides an alternative, and perhaps more suitable, approach for risk 
assessment while overcoming the limitations due to lack of empirical data. Consequently, taking advantage 
of the real-time aircraft health monitoring data not only enables a dynamic risk assessment capability but 
also helps alleviate the reliance on subjective data for component/system reliability rates. 


B. Off-Nominal Trajectory and Impact Point Prediction Module 


An important part of the URAF framework is the prediction of flight trajectories and/or ground impact 
point due to a severe off-nominal event. For the purposes of this discussion, an off-nominal event is defined 
as a significant deviation from the intended flight trajectory or a loss of control resulting in extreme vehicle 
attitudes (pitch, roll) exceeding the normal flight envelope. Thus, as a result of an off-nominal event, the risk 
of undesirable events such as impact with people or a stationary object (building) increases and is therefore 
a required consideration for risk assessment. 

Prediction of off-nominal trajectories presents significant challenges due to the many possible failure 
scenarios and the wide range of vehicle configurations that operate under the UTM framework. One of the 
objectives of the UTM project is the development of modeling and simulation methods that would allow 
prediction of flight dynamics behaviors and trajectories suitable for risk analysis as well as implementation in 
a real-time risk assessment tool.24-?® Due to practical considerations, low-order, generic modeling methods 
were sought since they offer the potential for rapid solutions and easier implementation. 

Currently little has been published regarding high-fidelity modeling of sUAS vehicles in off-nominal 
conditions. Modeling and simulation of multi-rotor vehicles is now the subject of research, but validated 
experimental data is limited. To date, trajectory prediction has been limited to point-mass or ballistics 
methods, which are generally applicable only to complete power failure or degraded control configurations. 
Lum et al.® simulated a large number of failures and observed the dynamics of the powerless aircraft to 
determine the impact zones and areas from these failures. The research concluded that vast majority of 
crashes occur close to the initial failure location. Similarly, Aalmoes et al.® attempted to represent the 
potential impact area as a function of the aircraft weight, speed, direction and maximum glide distance. 
Recent research by Foster & Hartman?* provided simulation predictions of multirotor trajectories during 
propulsion failures. The research results showed the prediction of a tumbling mode during abrupt motor 
failure and the resulting near-ballistic trajectory. In addition, a cascading power failure was shown to cause 
erratic transition dynamics as the vehicle entered into an out-of-control descent. 

One example of real-time trajectory prediction for sUAS vehicles is the flight termination system used 
for the NASA AirSTAR flight test facility.2”7 In the event of a flight anomaly requiring termination, the 
system was designed to put the vehicle in a sustained high angle of attack/high drag condition using pro- 
spin controls in order to create a near-vertical but predictable trajectory. An Impact Point Prediction (IPP) 
algorithm was developed to estimate this trajectory from initiation of termination to ground impact, based 
on the predicted rate of descent and known winds. An additional feature was an estimation of the transition 
dynamics from forward flight to the sustained descent. The algorithm was designed to provide impact 
distance accuracy of 10% of the initial altitude at termination. The IPP algorithm was implemented in a 
real-time Monte Carlo fashion in which important parameters such as total drag coefficient, wind direction 
and speed, and transition parameters were used as dispersion parameters. The 2-sigma potential impact 
area was simply computed from the intersection of the trajectory profiles with the known ground elevation. 
A limitation of this method is the assumption that the vehicle can achieve and maintain the high-drag 
condition as well as assumptions regarding the transition dynamics. Regardless of the method of choice, a 
high accuracy prediction of the impact point plays a crucial role in predicting risks to bystanders via the 
Casualty Estimation Module (discussed in the next section). 
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C. Casualty Estimation Module 


Following the estimation of the impact point and potential impact area, the next step in evaluating the sUAS 
risk is to determine the effects of the aircraft crash on the population below. Although financial impacts of 
UAS crashes can be substantial,° (e.g., UAS replacement value, repair of damage to property, environmental 
cleanup costs, etc.) within the URAF methodology only the human safety or loss of life was considered as 
the risk assessment metric. 

As previously discussed, a large number of studies have estimated risks associated with UAS operations 
over populated areas. These studies considered several parameters to estimate casualties. These parameters 
include population density and distribution, sheltering effects, casualty impact area (also called debris field 
or risk exposure area) and impacts of kinetic energy on casualty severity given the aircraft impact. Melynk 
et al.'° provided a through comparison of various approaches and examined the nuances and assumptions 
for these parameters. The comparison revealed that one of the most critical but varying parameters was 
related to the estimation of casualty area. The casualty area (or the hypothetical area where casualties are 
expected following an aircraft impact) along with the population density parameter is used to determine the 
number of people exposed to risk on the ground. 

Based on a literature review of well-known metrics, expected number of casualties, often referred as E,, 
was selected as the metric used to quantify the risk to bystanders in the URAF methodology. From a decision 
making perspective, the expected number of casualties is an appealing metric because it provides a single 
measure of safety and can be adjusted to provide casualty estimates given a specific scenario (conditional 
expectation). This metric has also been used widely in the range safety analysis of launch and reentry 
vehicles,?°:?° and thus it is well understood. A simple expression of the expected number of casualties given 
that an object impacts a certain location takes the following form: 


Ee = ppopAc (1) 


where Ppop represents the population density and A, represents the casualty area. This expression highlights 
the importance of estimating population density and the casualty area for even the simplest version of the 
expected number of casualties equation. Population density estimates are often derived from census data, 
while the casualty area is often a function of the vehicle shape and the impact trajectory. The use of the E, 
formulation along with the consideration of casualty area and sheltering effects are discussed in more detail 
in Section ITI.D. 


D. Risk Construct 


In order to determine and visualize the flight risk, the levels and boundaries of probability of mishap oc- 
currence as well as severities of the mishap need to be established. The modeled mishap likelihood and 
associated severities are combined to reflect the resultant risk, which is represented using a risk matrix, orig- 
inally introduced in Military Standard (MIL-STD) 882.°° The rows in the matrix reflect mishap likelihood 
and the columns provide severity categories where intersection of the two signifies the associated risk of the 
failure or mishap. The URAF methodology uses a modified version of the risk matrix developed for un- 
manned aircraft systems, which was cited within FAA’s Safety Management System Manual,*! to establish 
and visualize real-time UAS flight risks. 


1. Undesirable Event Likelihood 


Likelihood is defined as the estimated probability or frequency, in quantitative or qualitative terms, of a 
hazards effect or outcome.?! In order to express likelihood quantitatively, the assessment of failure rates 
is performed individually for each failure condition or mishap upon the basis of relevant experience, which 
can be acquired through a combination of data from testing, modeling and simulation, expert judgement, 
and structured analysis techniques.!®3° Barr et al.?? suggests employing allowable small aircraft hazard 
probabilities mandated by FAR Part 23 requirements; however, the range of allowable hazard probabilities 
is dependent upon the aircraft and the allowable risk levels and is still the subject of research. 

Given that the URAF methodology outputs mishap likelihood values derived from aircraft telemetry data, 
the authors propose employing these probability values instead of approximating failure rates per flight hour. 
Based on simulated case studies and data acquired from UAS hazard identification efforts, a modified version 
of the likelihood scale given by Haimes®° was adopted for the current iteration of the framework (Table 1). It 
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is important to note that the accepted likelihood values require continuous revision and refinement as more 
operational data associated with sUAS flight risks become available in the future. 


Table 1. Undesirable Event Probability Levels. 


Improbable | Remote | Probable | Frequent 
0< Pur < 0.01 | 0.01 < Poe < 01 | 0.1< Por <05|)05< Pur <1 


2. Undesirable Event Consequences 


Similar to the likelihood discussion above, the consequences of undesirable events need to be determined 
and ranked in order to construct the risk reference system for URAF. The FAA Safety Management System 
(SMS) manual defines severity as the consequence or impact of a hazard’s effect or outcome in terms of 
degree or harm. Consequently, the severity scales, often described qualitatively and quantitatively, range 
from minimal discomfort to those on the ground to fatality or fatal injury to bystandersbased on type of 
hazards.'*:34_ The mishap severity classification and definitions given in FAA’s SMS manual were adopted 
and modified to be used with URAF methodology. 

The estimated consequence metrics involve a combination of E, and kinetic energy (KE). This combina- 
tion takes place by discretizing KE and E,. Weights are then associated with each discretized level and the 
product of the KE weight and E, weight provides the final consequence level. The weights associated with 
KE and E, are: 


e Low KE = 1 

e High KE = 2 
e0< E < 0.1 =0; 
e01< E, < 0.5 = 1; and 
e05< Fo =2 


Using the product of the weights (or severity indices), the undesirable event consequences were classified 
as given in Table 2. For instance, Major category can be caused by both high KE coupled with 0.1 < E, < 
0.5, and low KE and 0.5 < FE. < 1 combinations. It is important to note that the descriptions provided in 
Table 2 aim to tackle the main concerns in the given severity index. The probability of fatality as a function 
of kinetic energy (KE) has been extensively covered in the literature+”19:!8 and was not within the scope of 
the URAF framework. For that reason, the consequence scale delineates between high and low KE based on 
the threshold determined by the method of choice. Similarly, the weight ranges can be adjusted according 
to mishap characteristics. 


Table 2. Undesirable Event Consequences. 


Minimal Minor Major Catastrophic 
Discomfort to those | Non-serious injury to Serious injury to Fatality or fatal injury 
on the ground people on the ground | people on the ground | to people on the ground 
Severity Index: [0] Severity Index: [1] Severity Index: [2] Severity Index: [4] 


38. URAF Risk Assessment Matrix 


As previously discussed, a risk matrix is used to classify and assess the risk with known likelihood and 
severity. The minimum acceptable level of risk is determined by severity categories; the FAA Risk matrix?! 
contains three risk categories (low, medium, and high), however two-level? or five-level?* (high, serious, 
medium, low, and eliminated) scales can also be employed based on the context. Based on the likelihood 
and severity scales discussed above, the risk matrix given in Table 3 was used in URAF. Based on the 
matrix mapping, mishaps are assigned High (highlighted red, unacceptable risk), Medium (highlighted yellow, 
minimum acceptable risk), and Low (highlighted green, acceptable without restrictions).3! It is important 
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to note that an improbable failure with catastrophic consequences was considered High risk due to numerous 
single point failure sources present with sUAS operations. 


Table 3. URAF Risk Matrix. 


: Minimal Minor Major Catastrophic 
Severity —> 


Likelihood | 


Severity Severity Severity Severity 
Index: [0] Index: [1] Index: [2] Index: [4] 


Frequent 

0.5 < Pra <1 
Probable 

0.1 < Por < 0.5 
Remote 

0.01 < Pur < 0.1 

Improbable 
0 < Pur < 0.01 


III. Case Study 


As part of UTM TCL demonstration efforts, a series of flights are being planned to showcase technologies 
developed by NASA LaRC UTM Safety team. The flights are scheduled to be performed during the Summer 
of 2017. In order to demonstrate the framework proposed in Section II, representative TCL flight log data 
was simulated for use as a case study. The following sections present the details and URAF methodology 
outputs for this flight. 


A. Flight Mission and Assumptions 


The simulation used a DJI $1000°? octocopter that was programmed to follow an autonomous trajectory 
over NASA LaRC. The case study was designed to accomplish simulated safety objectives including corona 
detection at a LaRC power substation, a pipeline inspection, an encounter with a non-cooperative aircraft, 
and a low altitude flight in an urban setting. A tool that was developed based on the URAF methodology 
was enabled on the ground control station computer to provide real-time flight risk due to a crash following 
an unpowered descent, which was considered as the undesirable event. The ground control station ran the 
Mission Planner open source software that holds the Python script for exchanging data between programs. 
The Mission Planner software received MAVLINK messages via the telemetry link from the aircraft equipped 
with PixHawk autopilot. The aircraft state information was then processed with MATLAB and Python 
scripts to execute both the Hugin BBN model and off-nominal trajectory and impact point estimation 
subroutines. Once the impact point was calculated, the aircraft state data and the impact point data were 
sent to APM Planner v2.0°3 software for visualization to the user. To demonstrate varying levels of casualty 
risks, the simulation included system failures while the aircraft was flying over people located out in the 
open. The specific scenario variables are discussed in Section III.E. 

The goal of the first iteration software given in this case study was to demonstrate the feasibility of the 
proposed URAF methodology. Consequently, the authors primarily focused on linking the aircraft provided 
data with the software suite and obtaining cohesion among the aforementioned modules. For that reason, 
several assumptions were made to ensure the minimal capability was achievable within an easily observable 
context. A summary of these assumptions along with planned capability improvements are given in Table 4. 
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Table 4. Case Study Assumptions and Planned Improvements. 


eetieed Case Study Scope & Assumptions | Planned Improvements 
Components 
Aircraft -Aircraft Health Monitoring parameters | -Implementation of array of sensors 
Telemetry limited to GPS, Telemetry Health, and | providing comprehensive aircraft 
Downlink Power System Status health data (servo and thrust response) 
-Static population density derived from 
; Pa es be e -Near real-time population distribution 
Population building characteristics and occupancy ; . 
: ae j . and density via census data augmented 
Density -Capability to customize population ae 
et ae by cellular network activity 
distribution 
-Reliable wind profile data obtained 
Environmental | -Preloaded wind profiles obtained fe 
ie by on-board sensors or UTM data 
Parameters from aviation weather sources : : 
service providers 
-Enhanced aircraft system and failure 
oes nists -Low-fidelity aircraft systems model mode representation populated with 
Probabilistic ? . . : 
: populated with arbitrary data real-time, operational and SME data 
Graphical Neue 
Model -No human factors models (operator, -Organizational and human factors 
ode 


maintenance adverse conditions) 


modeling to estimate operator and 
maintenance related issues 


Trajectory & 
Impact Point 


-Single mishap profile (unpowered 
descent /terminated flight) 
-Descent rate equivalent to terminal 


-Multitude of off-nominal conditions 
(mid-air collision, uncontrolled descent, 
partial aircraft control, propulsion and 


ae velocity power failure) 
Prediction P . ; ; ‘ : 
-2-sigma impact point uncertainty -Dedicated trajectory model for each 
obtained by Monte Carlo analysis considered mishap types 
-Casualty area calculation assumes 
vehicle glide -Inclusion of various secondary impact 
Casualty en tee ere ne 
Estimati -Uniform population distribution within | effects coupled with augmented 
stimation 
each grid cell population density and distribution 
Module 


-No secondary impact effects (e.g., fire 
casualties due to building damage, etc.) 


information 


B. Bayesian Belief Network Model 


As previously discussed, BBN modeling using Hugin Developer software was implemented as the probabilistic 
modeling method to estimate mishap likelihoods based on real-time aircraft component status. In order to 
do so, a simplistic model given in Fig. 2 was developed. The BBN model continuously updates the input 
values for GPS Count, GPS Status, Remaining Battery, Battery Voltage, Telemetry Health, and Wind Speed 
dependent upon the aircraft altitude. These values are then used to calculate the probability of associated 
system failures. The model outputs are used as both a decision making aid and as a precursor for an 
imminent mishap to inform the ground control station operator. The Return to Base (RTB) node provides 
the threshold value for executing RTB action based on minimum acceptable Navigation capability and Lost 
Link status. The CPT for the RTB node is populated to behave like a switch to inform the operator that 
the RTB execution is preferable due to evidence that the navigation system and/or command and control 
link may be operating below acceptable limits. The Unpowered Descent represents the modeled undesirable 
event based on the combined likelihood of power system failure and degraded controllability. The probability 
output from the BBN model is used in the likelihood axis of the risk matrix discussed in Section III.D. The 
aircraft mishap modeling and data population processes were previously discussed in detail by Ancel et al.?! 
and Barr et al.?? 
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Figure 2. BBN Model. 


C. Trajectory and Impact Point Prediction 


This case study used the AirSTAR Impact Point Prediction algorithm?’ previously discussed in Section 
II.C. Given that the purpose of this effort was to demonstrate the conceptual implementation of a real-time 
trajectory prediction algorithm, a high-fidelity representation of the actual off-nominal trajectory was not 
considered. The development of more generic and robust methods for numerous types of failures is being 
contemplated for future implementations of the URAF methodology. 

The inputs to the algorithm included the estimated average drag coefficient for a multi-rotor vehicle in 
tumbling motion (Cp), air density (p), the reference vehicle area (S), and the vehicle weight (W). The 
altitude rate of change (h) was estimated with Eq.(2) where the rate of descent was assumed to be the the 
vehicle airspeed in a 1g, near-vertical flight path, which corresponds to terminal velocity. As Dalamagkidis 
et al.1? suggested, the use of terminal velocity (or maximum operating velocity increased by 40% as an 
alternative value) is probably an over-conservative estimation for small UAS flying at low altitude. However, 
for this case study the assumption was used for estimating kinetic energy on impact. 


—_—— W/S 
v= ] $pCp 2) 


The transition dynamics were modeled in an ad hoc fashion that assumed a level transition to a constant 
rate of descent and a first-order decay in forward ground speed. Table 5 describes the dispersion parameters 
and associated dispersion values to obtain a 2-sigma potential impact area. The values shown for transition 
time, airspeed decay time constant, and drag coefficient were based on preliminary estimates from Foster & 
Hartman** and Cunningham et. al.?” 
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Table 5. Dispersion Parameters for IPP Monte Carlo Algorithm. 


Dispersion Parameter Nominal Value | Dispersion 
Transition time (sec) 2.5 +/- 0.5 
Inertial airspeed decay time constant (sec) 2.5 +/-1 
Wind direction (deg) Measured +/- 10 
Wind speed (knots) Measured +/-5 
Drag coefficient 0.4 +/- 0.2 


D. Expected Casualty Calculation 


The risk due to a sUAS crash on the general population was quantified by using the expected number of 
casualties, as discussed in Section IT.C. In order to do so, the EZ, components, population density, ppop, and 
Casualty Area, A,, need to be modeled. 


1. LaRC Population Density 


Unlike traditional manned aircraft, probability of injury or fatality due to sUAS crash is highly dependent 
on the operating location of the aircraft.° Within the context of the case study, the population density of 
NASA LaRC within the vicinity of the planned trajectory was modeled to quantify the expected number 
of casualties in case of a crash. In order to do so, an estimated number of occupants for each building was 
documented. Next, the latitude and longitude coordinates were obtained from the NASA LaRC Geographic 
Information System (GIS) database. The information was then combined to estimate population density in 
and around each building, in parking lots, and on walkways since the model allows assigning a percentage 
of population to be out in the open. Figure 3 shows the population density model of the buildings around 
the planned aircraft trajectory. 
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(a) Modeled Population Density Around the Trajec- (b) Aircraft Trajectory and Satellite View 
tory. of the Area. (Source: NASA GIS Team) 


Figure 3. NASA LaRC Population Density Model and Planned Trajectory. 


2. Potential Impact and Casualty Areas 


As there are no passengers to consider, the determination of the potential impact and the casualty areas 
associated with a failed UAS play a crucial role in determining the expected number of casualties. Figure 4 
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illustrates the concept of the casualty and potential impact areas. The potential impact area is defined by 
any location on the ground where the aircraft could crash. In this case study, the IPP module was employed 
to determine the size and location of the potential impact area considering trajectory and presence of wind. 
Additionally, for simplicity, the potential impact area was assumed to have a uniform distribution (i.e., the 
vehicle will impact any location inside the circle with the same probability). The casualty area is depicted 
with a red rectangle within the potential impact area and it is assumed that any person inside is considered 
a casualty given that the vehicle has enough kinetic energy on impact. The casualty area considers the 
aircraft wingspan, glide/impact angle, and representative characteristics of a person, as discussed in the 
next sections. 


{| 


Wind Direction 


Casualty 
Area (A,) 


Potential AG 


Impact Area 


Figure 4. Representative Potential Impact and Casualty Areas. (Source: NASA GIS Team) 


3. Sheltering Effects 


The expected casualty calculation takes into consideration people in different sheltering categories. Previous 
work done in the range safety analysis community was leveraged to provide vulnerability models to estimate 
risks to people inside buildings. The Range Safety Group?® proposed employing the building classes given 
in Table 6 for their vulnerability models. The vulnerability models define the relationship between object 
(e.g., UAS) mass, ballistic coefficient, and the effective casualty area. Figure 5 provides the model used for 
a class B roof. A more detailed discussion regarding the development and use of the vulnerability models 
can be found in the FAA Flight Safety Analysis Handbook* and in Safety Design for Operations.*° 


Table 6. Building Classes. 


Roof Class | Description 


A -Mobile homes and trailers 


-Temporary office trailers 


-Single family dwellings 

-Duplex and fourplex residential dwellings 
-Small condominiums and townhouses 
-Small apartment buildings 


-Small retail commercial buildings (gas 
Cc stations, stores, restaurants, strip malls) 
-Small office and medical office buildings 


-Public buildings (large office buildings, 
shopping malls, and apartment complexes, hotels, etc). 
- Warehouses 


-Manufacturing plants 
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Figure 5. Representative Vulnerability Model for Class B Roof Type. 


4. Casualty Estimation 


Equation (1) needs to be expanded to take into consideration the probability of impact and different sheltering 
categories. Given the population density map for NASA LaRC, we discretized the potential impact area into 
N grid cells. The expected number of casualties for the jth sheltering category is given by: 


N 
Ee; = S- Pp, Ac; Ppop, (3) 
kel 


where Py, is the probability of impacting the k'” grid cell. A, and Ppop, are the casualty area and population 
density for the j” sheltering category. Finally, the expected number of casualties given that there is a failure 


resulting in a crash is given by: 
For = S- Ei, (4) 
j=l 


The vulnerability models discussed in the previous section provide the casualty area given the mass of 
the vehicle and ballistic coefficient. The expression used to compute the casualty area for people in the open 
is given by: 


H, 
Ac = span 2R Luay —. 2R 5 
(Wspan + 2Rp)( + tany + 2Rp) (5) 
where Wspan represents the wing span, R, is the radius of a person, Duay is the length of the UAV, Hp is 
the height of a person, and y represents the glide angle. This expression has been previously introduced 


by Clothier et al.’and Lum et al.” Employing the methods above, the flight mission described in III.A was 
simulated. The results for this simulated mission are discussed in the next section. 


E. Case Study Details and Results 


The Mission Planner log file for the aforementioned flight mission was employed to simulate a predetermined 
list of scenarios in order to showcase the risk assessment software capabilities. The simulated scenario 
variables included power system glitches, presence of high winds, and a fire drill that caused the occupants 
to evacuate a building on the vehicle’s flight path. Figure 6 depicts the ground control station computer 
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representation of the software, indicating the UAS location, potential impact area, and assessed risk of 
unpowered descent considering the population density on the ground. 


{APM Planner v2.0.24 (LASLW16050584: 169,254.127.56/146.165.198. 184) (Sia) x) 


File Tool Widgets Help 


@ * ew : x { ]} (unknown | [_] 


LAT: LON: FIX: 3D ‘SATS: 254 HDOP:0 ZOOM: 19 


Bin, 


Likelihood 


(Gatien) Structures) Govome] (last Pos] Cache) (Clear wap] ‘Centre on UAV. Finished jptions 


Figure 6. Snapshot of the Ground Control Station Output. 


Figure 7 outlines the mishap probability, E, value, and resultant risk level for the duration of the simulated 
flight. The first failure (depleting battery capacity and lower battery voltage) was introduced around t=200 
seconds, which caused the unpowered descent probability to increase to 80% for around one minute. From 
t=260 seconds to t=580 seconds, the battery voltage and remaining battery leveled off at an acceptable 
level. Meanwhile, the EF, value stayed below 0.1 (severity index of [0]), which indicated that in the event 
of a crash, the consequences would be minimal irrespective of KE or mishap probability. For that reason, 
during the first simulated failure, the flight did not pose any risk to humans on the ground. 

The next simulated failure (at t=581 s) brought the remaining battery and battery voltage values to 
the limit of acceptable thresholds, causing the mishap likelihood to rise to 0.85. Additionally, the winds 
were substantially increased (25 m/s at 90 degrees), drifting the potential impact area towards a simulated 
outdoor activity (e.g, fire drill, outdoor gathering, etc.). This caused the FE, to rise gradually to over 0.8 
(severity index of [4]). The snapshot given in Fig. 6 represents the simulated failure, increased wind presence, 
and wind impact on aircraft impact point drift while the aircraft is flying along the trajectory at t=650. 
The combined effects of high probability mishap (unpowered descent due to power system failure), higher 
E, (increased population density without protection of sheltering), and the presence of winds (affecting the 
flight controllability, increasing the KE on impact, and causing the aircraft to drift towards populated area) 
quickly escalated the flight risk from low to medium to high as shown in Fig. 6. 
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Figure 7. Simulated Model Output, Expected Casualty/Mishap Probability /Risk levels vs. Time 


IV. Future Work 


A. Planned Improvements 


The primary focus of the case study was to assess the feasibility of the URAF methodology. The future 
work involves the implementation of planned improvements previously given Table 4. The authors believe 
that the most valuable improvement to the current software would be the inclusion of dynamic population 
distributions. Future research will explore opportunities associated with fusing the census demographic 
information with data provided by cellular network towers to obtain a high-resolution augmented population 
density and distribution capability. Also, by extracting additional aircraft health parameters, not only the 
dynamic mishap probability estimation can be drastically enhanced but also other failure modes and mishaps 
can be modeled. Similarly, the prediction of impact point accuracy can be improved by higher fidelity models 
like ballistics methods and better estimation of aerodynamic parameters via wind tunnel testing. 


B. Validation Efforts 


It is crucial to determine the accuracy of predictive models. However, validating the casualties associated 
with UAS crashes is impractical due to a lack of data on historical accidents. Although Belcastro et al.% 
reviewed a sizable database concerned with sUAS mishaps, any meaningful statistical inference is unlikely 
due to the vast variety of aircraft and operational contexts. The literature review indicated that Melnyk 
et al.!8 proposed employing bystander fatality rates for General Aviation (GA) accidents by adjusting the 
model parameters to reflect a typical GA aircraft. Their analysis suggested that model casualty results were 
representative of casualty rates obtained by historical data. To this point, the authors have not pursued 
validation of the URAF methodology given the demonstrated case study was solely used to gauge the 
feasibility of the approach. However, future versions featuring high-fidelity models will be evaluated against 
historical data, as demonstrated by Melynk et al. 


V. Conclusions 


As projected demand for unmanned aircraft operations increases, risk identification and assessment will 
continue to play an important role in developing the procedures and protocols necessary for widespread 
adoption of these systems. NASA’s UTM research initiative aims to enable low-altitude civil UAS operations 
while ensuring safety of the people in the air and on the ground. The framework presented in this paper 
provides guidance to develop a series of tools that are capable of providing real-time risk assessment for 
sUAS operations. Leveraging past third party risk assessment studies, the framework proposes the use 
of aircraft generated health monitoring data along with augmented population density and other dynamic 
environmental inputs to evaluate casualty risk and inform the operator of imminent failures. Besides serving 
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as a ground operator support role, the proposed real-time risk assessment capability can be embedded into 
UTM services or be used to evaluate mitigation strategies and manage contingencies. A simplified case study 
was used to demonstrate the feasibility of the concept. By extending the analysis to include other mishaps 
while taking advantage of higher-fidelity trajectory and casualty estimation methods, the framework has the 
potential to assist regulators in crafting requirements and applicable safety standards for large scale sUAS 
operations. 
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